Modeling voice calls to improve an outcome of a call between a representative and a customer

ABSTRACT

A call-modeling system models calls in real-time, with the goal of helping users, e.g., a sales representative and/or their managers, improve and/or guide the outcome of the calls. The call-modeling system generates real-time probabilities for possible outcomes of the conversation, as well as highlight specific on-call patterns, which may be either conducive or detrimental to a desired conversation outcome. The generated probabilities and highlighted patterns may be used by the sales representatives and/or their managers to either increase the probability of a desired outcome and/or optimize for call duration with a specific outcome.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/017,646 titled “MODELING VOICE CALLS TO IMPROVE AN OUTCOME OF A CALLBETWEEN A REPRESENTATIVE AND A CUSTOMER” filed Jun. 25, 2018, now U.S.Pat. No. 10,530,929, issued Jan. 7, 2020, which is acontinuation-in-part of U.S. patent application Ser. No. 15/168,675titled “MODELING VOICE CALLS TO IMPROVE AN OUTCOME OF A CALL BETWEEN AREPRESENTATIVE AND A CUSTOMER” filed May 31, 2016, now U.S. Pat. No.10,051,122, issued Aug. 14, 2018, which claims the benefit of U.S.Provisional Application Ser. No. 62/169,456 titled “MODELING VOICE CALLSTO IMPROVE AN OUTCOME OF A CALL BETWEEN A SALES REPRESENTATIVE AND ACUSTOMER” filed Jun. 1, 2015, and U.S. Provisional Application Ser. No.62/169,445 titled “COORDINATING VOICE CALLS BETWEEN SALESREPRESENTATIVES AND CUSTOMERS TO INFLUENCE AN OUTCOME OF THE CALL” filedJun. 1, 2015, all of which are incorporated herein by reference for allpurposes in their entirety.

BACKGROUND

With over 2.4 million non-retail inside sales representatives in theUnited States (U.S.) alone, millions of sales phone conversations aremade on a daily basis).^(i) However, except for rudimentary statisticsconcerning e.g., call length and spotted keywords and phrases, salesconversations are left largely unanalyzed, rendering their contentinaccessible to modeling, and precluding the ability to optimize themfor desired outcomes. Recent advances in automatic speech recognition(ASR) technologies, and specifically in large vocabulary continuousspeech recognition (LVCSR), are for the first time enablinghigh-accuracy automatic transcription of conversations. At the sametime, natural language processing (NLP) approaches to both topicmodeling and world-knowledge modeling, have become much more efficientdue to the availability of large, freely accessible natural languagecorpora (e.g., CommonCrawl), as well as freely available ontologies or“knowledge graphs” (e.g., DBpedia). Finally, recent research on affectidentification applying machine learning (ML) has been able tosuccessfully model subjective aspects of emotion and personality traitsas perceived by listeners.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a call-modeling system in which thedisclosed embodiments can be implemented.

FIG. 2 is a block diagram of a feature generation component of thecall-modeling system for extracting features from call data, consistentwith various embodiments.

FIG. 3 is a block diagram of a classifier component of the call-modelingsystem for generating classifiers, consistent with various embodiments.

FIG. 4 is a block diagram of a real-time analysis component of thecall-modeling system for generating on-call guidance for arepresentative during a call between the representative and a customer,consistent with various embodiments.

FIG. 5 is a flow diagram of a process for performing offline analysis ofconversations between participants, consistent with various embodiments.

FIG. 6 is a flow diagram of a process for modeling calls between theparticipants to generate on-call guidance, consistent with variousembodiments.

FIG. 7 is a block diagram of a processing system that can implementoperations of the disclosed embodiments.

DETAILED DESCRIPTION

Embodiments are disclosed for a call-modeling system for modelingconversations, e.g., voice conversations, in real time, with the goal ofhelping users, e.g., sales representatives and/or their managers, toimprove and/or guide the outcome of conversations with other users,e.g., customers. One such embodiment can model the calls based oncharacteristics of the conversation, e.g., voice of the representativesand/or the customers, and content of the conversation, with the goal ofpositively influencing the outcome of the call. The call-modeling systemcan generate real-time probabilities for possible outcomes of areal-time conversation, e.g., an ongoing conversation between a specificrepresentative and a customer, and generate specific on-call guidance,which may be either conducive or detrimental to a desired conversationoutcome. The generated probabilities and on-call guidance may be used bythe representatives and/or their managers to either increase theprobability of a desired outcome and/or optimize the conversation for aspecified duration if the predicted outcome is not going to be a desiredoutcome. For example, for renewing a magazine subscription, thecall-modeling system can generate an on-call guidance suggesting arepresentative to engage in a rapport building exercise with thecustomer if it is determined that doing so increases the chances of thecustomer renewing the membership by 45%. On the other hand, if thecall-modeling system predicts from the on-going conversation that thecustomer is not going to renew the subscription membership, then thecall-modeling system can suggest the representative to wrap up theconversation as soon as possible.

The call-modeling system can include (i) an offline analysis componentand (ii) a real-time analysis component. The offline analysis componentcan take as input conversations between a calling party, e.g., acustomer, and a called party, e.g., a representative, and process theconversations using multiple distinct components to generate multiplefeatures of the conversations. In some embodiments, the conversationscan be audio recordings of calls between called party and the callingparty (collectively referred to as “participants”). The features caninclude transcripts of audio recordings, vocabulary, semanticinformation of conversations, summarizations of utterances and variousnatural language entailments, voice signal associated features (e.g., aspeech rate, a speech volume, a tone, and a timber), emotions (e.g.,fear, anger, happiness, timidity, fatigue), personality traits (e.g.,trustworthiness, engagement, likeability, dominance, etc.), personalattributes (e.g., an age, an accent, and a gender),customer-representative pair attributes that indicate specificattributes associated with both the speakers that contribute to aspecified outcome (e.g., similarity of speech rate between therepresentative and the user, extrovert/introvert matching, or gender orage agreement).

In some embodiments, the audio recordings can be tagged with varioustags, e.g., a tag that indicates a trait (e.g., “extrovert”,“trustworthy voice”, “anxious”, etc.) of one or more of theparticipants, a tag that indicates a call outcome (e.g., “sales closed”,“sales failed”, or “follow-up call scheduled”), and/or a tag thatindicates “key moments” of a conversation. A “key moment” or a “moment”can be a specific event or an occurrence of a specified characteristicin the call. The moment can be any event or occurrence that is ofspecific interest for a specified application for which thecall-modeling system is being implemented. An administrator of thecall-modeling system can configure what events and/or occurrences in acall have to be identified as a moment. For example, a moment caninclude laughter, engagement, fast-talking, open-ended questions,objections, etc. in a conversation. The tags can be generatedautomatically by the call-modeling system, manually, e.g., by humanjudgment, or both. In some embodiments, the tags are generatedautomatically by the call-modeling system. The tag can include variousdetails, e.g., information regarding a moment, a time interval at whichthe moment occurred in the call, duration for which the moment lasted,information regarding the participants of the call, etc.

The moments can also be notified to and/or shared between theparticipants during an on-going conversation and/or after theconversation. For example, during a call between a user and arepresentative, the call-modeling system can analyze the call, identifythe moments in the conversation, and tag, notify and/or share themoments with the representative's manager, peers or other people. Theshared moments can be used for various purposes, e.g., for coaching therepresentatives in handling the calls to improve outcomes of the callsbased on various situations. The moments can be shared using variousmeans, e.g., via email, a chat application, or a file sharingapplication.

In some embodiments, the offline analysis component uses distinctcomponents to extract the features. The components can include anautomatic speech recognition (ASR) component, which can produce atranscription of the conversation, a natural language processing (NLP)component, which can extract semantic information (such as open-endedquestions asked, key objections, etc.) from the conversation, an affectcomponent, which can analyze the recording for emotional signals andpersonality traits (e.g., likeability and trustworthiness), and ametadata component, which can extract data regarding conversation flow(i.e., who spoke when, and how much silence and overlap occurred).

The offline analysis component can analyze the features to generate oneor more classifiers that indicate conversation outcomes, e.g., “salesclosed”, “sales failed.” Each of the classifiers indicates a specificoutcome and can include a set of features that contribute to thespecific outcome. The offline analysis component can generate multipleclassifiers for the same outcome; however, the multiple classifiers canhave distinct sets of features. In some embodiments, the offlineanalysis component can analyze the features using a machine learningalgorithm (e.g., a linear classifier, such as a support vector machine(SVM), or a non-linear algorithm, such as a deep neural network (DNN) orone of its variants) to generate the classifiers.

In some embodiments, the offline analysis component generates aclassifier for different time intervals or time windows of theconversations. For example, the offline analysis component can analyzethe extracted features for seconds 00:05-00:10 of a conversation,seconds 00:20-00:30, and minutes 1:00-2:00, and generate a classifierfor each of those time windows. The offline analysis component feeds theextracted features into the machine-learning algorithm to producemultiple classifiers corresponding to the time windows. The time windowscan be of varying lengths or fixed lengths. In some embodiments, theoffline analysis component can generate classifiers for other relativepositions of a conversation. For example, the offline analysis componentcan generate a classifier corresponding to an instance in theconversation, e.g., when a customer spoke for the first time in theconversation, and analyze features such as a pitch of the voice, a topicthe customer spoke about first, and the length of the customer's firsttalk, to generate the classifier.

The real-time analysis component uses the classifiers to model areal-time conversation, e.g., an ongoing call between a representativeand a customer, that helps the representative to increase a probabilityof a desired outcome of the conversation or optimize the conversationduration in case the real-time analysis component does not predict thedesired outcome. The real time analysis component receives real-timecall data of an ongoing conversation between the customer and arepresentative and analyzes the real-time call data to generate a set offeatures, e.g., using the offline analysis component as described above.The real-time analysis component can then feed the features to theclassifiers to generate probabilities of potential outcomes of the call.The real-time analysis component can use the classifiers with highestprediction powers to generate the probabilities of various potentialoutcomes. In some embodiments, the real-time analysis component measuresthe prediction powers of the classifiers using an F-score, which, instatistical analysis, is a (possibly weighted) harmonic mean ofprecision and recall.

The real-time analysis component feeds the extracted features into theclassifiers with high F-scores to generate probabilities of possibleoutcomes. Based on the probabilities, the real-time analysis componentcan also generate on-call guidance, which encourages the representativeand/or their managers to modify, desist or persist with a specifiedon-call behavior to increase or decrease the probability of one of thepossible outcomes, e.g., a desired outcome such as closing a sale. Insome embodiments, the on-call guidance includes a set of suggestedfeatures and their values to be adopted, desisted or persisted with bythe representative. For example, the on-call guidance can includeinstructions for the representative to change the rate of speech (e.g.,speak slower), use specific key words, or pose more open-ended questionsto the customer.

In some embodiments, the on-call guidance can change as the callprogresses, e.g., based on the classifiers that are relevant to the callat that particular time of the conversation. For example, during thefirst two minutes of the call, a classifier that corresponds to thefirst two minutes of the call may be used to generate the on-callguidance such as instructing the representative to pose open-endedquestions to the customer, and then in the third minute, a classifierthat corresponds to the third minute of the call may be used to revisethe on-call guidance, e.g., suggest to the representative to adjust thespeech rate to match with that of the customer.

Additionally, if according to the classifiers, the real-time analysiscomponent predicts the conversation to fail, the on-call guidance maysuggest to the representative to quickly wrap up the call in order tospare the representative's time. The on-call guidance of the real-timeanalysis module may be presented on-screen or via any other interface(e.g., voice instructions given through an ear piece) to therepresentative and/or the manager. The embodiments can produce real-timeprobabilities of various outcomes of the conversations, enabling livecoaching that can help the representatives in improving the outcomes ofthe conversations in real-time.

While the application describes the call data as being an audiorecording of a conversation between the participants, it is notrestricted to an audio recording. The call data and/or real-time calldata can be of a conversation that is any of telephone based, Voice overInternet Protocol (VoIP) based, video conference based, Virtual Reality(VR) based, Augmented Reality (AR) based, electronic mail (e-mail)based, or based on any online meetings, collaborations or interactions.The call data and/or real-time call data can also be of a conversationbetween two or more speakers physically located in the same room.

Turning now to FIG. 1, FIG. 1 is a block diagram of a call-modelingsystem 100 in which the disclosed embodiments can be implemented. Thecall-modeling system 100 includes an offline analysis component 110 anda real-time analysis component 130. The offline analysis component 110can take as input historical call data 105, which includes conversationsbetween participants, e.g., audio recordings of calls betweenrepresentatives and customers, and process the call data 105 usingmultiple components to generate features 115 of the conversations, andclassifiers 120.

The offline analysis component 110 includes a feature generationcomponent 111 that generates features 115 by analyzing the call data 105using various techniques, e.g., ASR, NLP. The features 115 can includetranscripts of audio recordings, vocabulary, semantic information ofconversations, summarizations of utterances and various natural languageentailments, voice signal associated features (e.g., speech rate, speechvolume, tone, and timber), emotions (e.g., fear, anger, happiness,timidity, fatigue), personality traits (e.g., trustworthiness,engagement, likeability, dominance, etc.), personal attributes (e.g.,age, accent, and gender), and inter-speaker attributes that indicate acomparison between both the speakers (e.g., similarity of speech ratebetween the representative and the user, extrovert/introvert matching,or gender or age agreement).

The classifier component 112 analyzes the features 115 using varioustechniques, e.g., machine learning algorithms such as SVM, DNN, togenerate the classifiers 120. The classifiers 120 indicate conversationoutcomes, e.g., “sales closed”, “sales failed,” “probability ofrecommending to a friend,” a measure of “customer satisfaction,” and NetPromoter Score (NPS). An outcome can have binary values, e.g., “yes/no”,“high/low”, or non-binary values, e.g., a probability score, enumeratedvalues like “low, average, medium, high, very high,” values on a scaleof 0-10, etc. For example, an outcome such as customer satisfaction canbe measured using binary values such as “low/high”, or using non-binaryvalues, such as a scale of 0-10, enumerated values. Each of theclassifiers indicates a specific outcome, a probability of the specifiedoutcome and can include a set of the features that contributed to thespecific outcome. For example, in a sales call for renewing a magazinesubscription, a classifier “C1” can indicate that when laughter by acustomer and two open-ended questions from the representative areregistered, there is a high chance, e.g., 83%, of renewal.

In some embodiments, the classifier component 112 generates differentclassifiers for different time windows of the conversations. Forexample, the classifier component 112 generates a classifier “C1” forthe first two minutes of the conversations and a classifier “C2” for athird minute of the conversations. The classifier “C1” based on thefirst two minutes of the conversation can indicate that when laughter bya customer and two open-ended questions from the representative isregistered, there is a high chance, e.g., 83%, of renewal. Theclassifier “C2” based on the third minute of the conversation canindicate that when a competitor magazine or the key-phrase “read online”is used, the renewal chances drop to 10%, all of which can occur ifcustomer's speech rate drops below three words per second. Some of theclassifiers include features for inter-speaker attributes that indicatea comparison between the speakers that contribute to a specified outcome(e.g., similarity of speech rate between the representative and theuser, extrovert/introvert matching, or gender or age agreement).

Note that the features, when extracted from the conversations, are justattributes, i.e., without values, such as “gender”, “age”, “speechrate”, etc., and the classifier determines what values of the featuresinfluence a particular outcome of the call. The classifiers 120 can begenerated in various formats and is not limited to the above illustratedexample format. The classifier component 112 can generate multipleclassifiers for the same outcome; however, the multiple classifiers canhave distinct sets of features. Further, as described above, theclassifier component 112 can generate different classifiers fordifferent time windows of the conversation. The offline analysiscomponent 110 can store the features 115 and the classifiers 120 in astorage system 125.

The call-modeling system 100 includes a real-time analysis component 130that uses the classifiers 120 to generate on-call guidance for bothinbound and outbound calls that will help the representative optimizethe call for a desired outcome, or optimize the call duration if thedesired outcome is not predicted (i.e., very low chances of the desiredoutcome are predicted). The real-time analysis component 130 receivesreal-time call data 150 of an ongoing conversation between a customerand a representative and analyzes the real-time call data 150 togenerate a set of features, e.g., call features 135, for the ongoingconversation using a feature generation component 113. In someembodiments, the feature generation component 113 is similar to or thesame as the feature generation component 111. The feature generationcomponent 113 generates the call features 135 based on the real-timecall data 150, e.g., as described above with respect to the featuregeneration component 111. The real-time call data 150 can be anearly-stage or initial conversation between the customer and therepresentative.

After the call features 135 are generated, a classifier component 114,which, in some embodiments, is the same as, or similar to the classifiercomponent 112, inputs the call features 135 to the classifiers 120 todetermine a set of classifiers 140 that predict possible outcomes of thecall based on the call features 135. Each of the set of classifiers 140indicates a specified outcome of the call and an associated probabilityof the corresponding outcome. In some embodiments, the classifiercomponent 114 chooses classifiers that have the highest predictionpower, which can be measured using an F-score, as the set of classifiers140. After the set of classifiers 140 are determined, a call-modelingcomponent 116 generates an on-call guidance 145 that includes real-timeprobabilities of possible outcomes of the call as indicated by the setof classifiers 140. The call-modeling component 116 can further analyzethe set of classifiers 140 to determine features that have highprediction power, e.g., prediction power exceeding a specifiedthreshold, for predicting a desired outcome, and include those featuresand values associated with those features in the on-call guidance 145.The on-call guidance 145 notifies the representative to adopt, desist orpersist with an on-call behavior consistent with those features toachieve the desired outcome, or to increase the probability of achievingthe desired outcome. If the set of classifiers 140 predict that thedesired outcome may not be achieved, the call-modeling component 116 maysuggest, in the on-call guidance 145, that the representative wrap upthe call.

The call data 105 can be in various formats, e.g., audio recordings,transcripts of audio recordings, online chat conversations. Similarly,the real-time call data 150 can be in various formats, e.g., real-timeaudio stream of the call, a chat transcript of an ongoing conversationin an online chat application. Further, the real-time call data 150,which can include an initial or early stage conversation, can be aconversation between the customer and an automated machine, e.g., aninteractive voice response (IVR) system, or a representative forgathering preliminary information from the customer that can be usefulfor generating the on-call guidance.

In some embodiments, the call-modeling system 100 includes a search toolthat empowers a user to explore various aspects of a conversation. Forexample, the search tool allows the user to search for anything thatcame up on the call, e.g., both linguistic and meta-linguistic. Thesearch tool can be used to further analyze the conversation, extractappropriate features and use them to improve the classifiers inpredicting the outcome of the calls. For example, the search tool can beused to find calls that registered a laughter from the customer, callsin which the customer spoke for the first time after a specified numberof minutes, calls in which the customer sounded angry, calls in whichcustomer mentioned competitors, calls in which the representativesengaged in rapport building, calls in which the representative modulatedspeech rates at various instances of the call, calls in which short oropen-ended questions were asked at a high frequency, or any combinationof the above.

FIG. 2 is a block diagram of a feature generation component of FIG. 1for extracting features from call data, consistent with variousembodiments. In some embodiments, the feature generation component 111includes an ASR component 210, an NLP component 225, an affect component215 and a metadata component 220. The ASR component 210 can analyze calldata 205, e.g., a voice recording, and produce a transcription,vocabulary, and a language model of the conversation. The NLP component225 can extract semantic information, such as key objection handlingresponses, from the output of the ASR component 210. The affectcomponent 215 can analyze the call data 205 for emotional signals andpersonality traits (e.g., likeability, extroversion/introversion, andtrustworthiness) as well as general personal attributes such as gender,age, and accent of the participants. The metadata component 220 canextract data regarding conversation flow (e.g., who spoke when, and howmuch silence and overlap occurred). In some embodiments, the abovecomponents can process the call data 105 in parallel. The output of thecomponents can be generated as features 115 of the conversations, whichcan be analyzed to determine outcomes of the conversations.

The ASR component 210 may be tuned for specific applications, e.g., forsales calls. The features produced by the ASR component 210 may includefull transcripts, vocabularies, statistical language models (e.g.,transition probabilities), histograms of word occurrences (“bag ofwords”), weighted histograms (where words are weighted according totheir contextual salience, using e.g., a Term Frequency-Inverse DocumentFrequency (TF-IDF) scheme), n-best results, or any other data availablefrom the component's lattice, such as phoneme time-stamps, etc. The ASRcomponent 210 may also be used to extract meta-linguistic features suchas laughter, hesitation, gasping, background noise, etc. The ASRfeatures can be extracted separately for the representative and thecustomer, and may be recorded separately for multiple speakers on eachside of the conversation.

The NLP component 225 processes the text to produce various semanticfeatures, e.g., identification of topics, identification of open-endedquestions, identification of objections and their correlation withspecific questions, named entity recognition (NER), identification ofrelations between entities, identification of competitors and/orproducts, identification of key phrases and keywords (eitherpredetermined, or identified using salience heuristics such as TF-IDF),etc. Additional features that may be extracted by the NLP component 225can be summarizations of utterances and various natural languageentailments. The NLP features can be extracted separately for therepresentative and the customer, and may be recorded separately formultiple speakers on each side of the conversation.

The affect component 215 can extract low-level features and high-levelfeatures. The low-level features can refer to the voice signal itselfand can include features such as speech rate, speech volume, tone,timber, range of pitch, as well as any statistical data over suchfeatures (e.g., maximal speech rate, mean volume, duration of speechover given pitch, standard deviation of pitch range, etc.). Thehigh-level features can refer to learned abstractions and can includeidentified emotions (e.g., fear, anger, happiness, timidity, fatigue,etc.) as well as perceived personality traits (e.g., trustworthiness,engagement, likeability, dominance, etc.) and perceived or absolutepersonal attributes such as age, accent, and gender. Emotionidentification, personality trait identification, and personalattributes, may be trained independently to produce models incorporatedby the affect component, or trained using the human judgment tagsoptionally provided to the offline analysis component. In someembodiments, the affect component 215 can also extract features, such asa speaker engagement metric (“wow” metric), which measures how engaged aparticipant was in the conversation, e.g., based on the usage ofvocabulary, rate of speech, pitch change. For example, the usage ofphrase “Oh! cool” can indicate a higher degree of engagement than thephrase “cool!”. In another example, the same phrase but said indifferent pitches or pitch ranges can indicate different degrees ofengagement. All features extracted by the affect component 215 may ormay not include a corresponding confidence level, which can be used inmodeling outcomes. The affect features can be extracted separately forthe representative and the customer, and may be recorded separately formultiple speakers on each side of the conversation.

The metadata component 220 can measure conversation flow, includingspeaker diarisation (e.g., which speaker spoke when and for how long),silence times and duration, as well as overlap of two or more speakersin addition to other metadata such as time of day call was placed,geographical destination of call and known gender and age ofparticipants. The data extracted with the metadata component 220 may becollected separately for multiple speakers on each side of theconversation, or pooled together for representative and customer sides,respectively.

All components may extract features for a group of representatives, asingle representative and/or a customer, including multiple parties oneither side, and may be customized to optimize feature extractionaccordingly. In addition, the features 115 may be extracted on therepresentative's recording alone, on the customer's recording alone, oron both. The features 115 may also include comparisons between extractedattributes. For example, the affect component 215 may extract as afeature a mean difference between representative and customer's speechrates, or a maximum difference between representative and customer'sspeech pitches. Likewise, the ASR component 210 may extracttranscriptions and keywords both as a combined transcript and as twoseparate transcripts, and may be tuned with an acoustic or languagemodel specific to a group of representatives or an individualrepresentative. Similarly, the NLP component 225 may extract featuressuch as open-ended questions with or without the corresponding response.

In some embodiments, the feature generation component 111 can alsogenerate a set of features that indicate a blueprint of a conversation.The blueprint can represent a skeleton of the conversation and indicatea presence or absence of a particular aspect in the conversation. Forexample, the blueprint can include various features that indicatewhether the conversation included any agenda setting, rapport building,clarification questions, defining goals, setting expectations,mentioning of examples. The blueprint can also help in predictiveanalysis of the outcome of the calls, e.g., by the classifier component112.

FIG. 3 is a block diagram of the classifier component for generatingclassifiers, consistent with various embodiments. The example 300illustrates the classifier component 112 using the features 115extracted from the feature generation component 111 to generate a numberof classifiers, “C1”-“CN”. In some embodiments, the classifier component112 analyzes the features of a dedicated portion of the collectedrecordings, e.g., a training set, which is a subset of the entirerecordings available for analysis, to generate the classifiers 120. Eachof the classifiers 120 can have a value, e.g., an F-score, thatindicates a prediction power of the classifier for the specifiedoutcome. The higher the prediction power, the higher the probability ofachieving the specified outcome of the classifier based on the includedfeatures. In some embodiments, the prediction power is determined byrunning the classifiers 120 on, e.g., a portion of call recordings thatis not yet analyzed, e.g., a test set, and computing the respectiveF-score.

The classifiers 120 may be further analyzed to determine what featurescarry the largest prediction power, e.g., speech rate early in theconversation, occurrence of first interrupt by customer, names ofcompetitors mentioned, or number of open questions thoughtfullyanswered, and a subset of these classifiers that have features with thelargest prediction power can be used to generate the on-call guidance.

The conversation outcome depicted by the classifiers 120 can be anyconfigurable outcome, e.g., “sales closed”, “sales failed”, “demoscheduled”, “follow up requested,” NPS-like probability of recommendingto a friend, etc. In some embodiments, the features 115 extracted fromthe feature generation component 111 can be fed into a machine learningalgorithm (e.g., a linear classifier, such as a SVM, or a non-linearalgorithm, such as a DNN or one of its variants) to produce theclassifiers 120. The classifiers may be further analyzed to determinewhat features carry the largest prediction powers (e.g., similarity ofspeech rate, occurrence of first interrupt by customer,extrovert/introvert matching, or gender or age agreement.)

The classifier component 112 can generate multiple classifiers for thesame outcome. However, for a given outcome, different classifiers havedifferent features. For example, the classifier component 112 cangenerate a first classifier 305, “C1,” and a second classifier 310,“C2,” for a specified outcome, “O1.” However, the first classifier “C1”has a first set of features, e.g., features “f1”-“f3,” and the secondclassifier “C2” has a second set of features, e.g., features “f5”-“f8.”The features in different classifiers can have different predictionpowers and contribute to the specified outcome in different degrees.

Different classifiers may be built for a different number ofparticipants, and may consider multiple participants as a singleinterlocutor, or as distinct entities. Further, as described above, theclassifier component 112 can generate different classifiers fordifferent time intervals of a conversation. The classifier component 112analyzes the features 115 extracted from the feature generationcomponent 111 at various time intervals, e.g., seconds 00:05-00:10,seconds 00:20-00:30, minutes 1:00-2:00, covering the entire conversationduration, and generates one or more classifiers for each of those timeintervals. Each classifier can correspond to a specified time intervalof the conversation. For example, if “100” conversations are beinganalyzed, then the classifier component 112 can analyze first 5-20seconds each of the “100” conversations and generate one or moreclassifiers for all the conversations corresponding to the interval of5-20 seconds. Similarly, it can generate one or more classifierscorresponding to the 10-25 seconds interval. If more than one classifieris generated for a specified time interval, in some embodiments,different classifiers can have different outcomes, and in someembodiments, can have the same outcome; however, different classifierswill have different sets of features that contribute to thecorresponding outcome. In the example 300, classifiers C1 and C5correspond to a time window of seconds 00:05-00:20 of the conversationsanalyzed, and classifier C10 corresponds to minute 1:00-2:00 of theconversations.

The classifier 315, “C3,” includes an example set of features extractedfrom analyzing various sales calls. The classifier 315 corresponds tothe first two minutes of the conversations, and indicates that whenlaughter from the customer is registered and the representative greetsthe customer, indulges in rapport building and poses at least twoopen-ended questions, then there is a high chance, e.g., 83%, of renewalof a magazine subscription. The features and outcome of the classifier315 “C3” can be “f1→customer laughter=yes” “f2→greeting customer=yes,”“f3→rapport building=yes,” (“f4→open ended questions asked=yes,” and“f5→number of open ended questions asked=2”), “outcome=renewsubscription” “probability of outcome=83%.”

The classifiers 120 can be used by the real-time analysis component 130,e.g., as described at least with reference to FIG. 1 above and FIG. 4below, to generate an on-call guidance for representatives or bothinbound and outbound calls. FIG. 4 is a block diagram of the real-timeanalysis component of FIG. 1 for generating on-call guidance for arepresentative during a call between the representative and a customer,consistent with various embodiments. In some embodiments, the real-timeanalysis component 130 takes as input a live conversation stream, e.g.,real-time call data 150, between a representative 410 and a customer405, uses the feature generation component 113 to extract call features135, e.g., as described above at least with reference to FIGS. 1 and 3.

The classifier component 114 feeds the call features 135 into theclassifiers 120 generated by the offline analysis component 110 andselects a subset of the classifiers 120, e.g., a set of classifiers 140,that includes features that match with the call features 135 extractedfrom the live conversation stream. In some embodiments, the set ofclassifiers 140 chosen by the call-modeling component 116 are also theclassifiers that have high predictability power, e.g., as measured usingan F-score and that have an F-score exceeding a specified threshold.

The call-modeling component 116 then generates an on-call guidance 145,which includes information regarding real-time probabilities forspecific outcomes to which the set of classifiers 140 correspond. Theon-call guidance 145 may be used to notify the representative and/ortheir managers of the predicted outcome of the call. Additionally, thecall-modeling component 116 can further analyze the set of classifiers140 to determine classifiers that include features with the largestprediction powers, and present the values of those features in theon-call guidance 145 for suggesting the representative and/or theirmanagers to modify or persist with an on-call behavior consistent withthose features. For example, if one of the set of classifiers 140predicts that conversations with rapport building and several open-endedquestions being posed at the first few minutes of the conversation leadto favorable outcomes, the call-modeling component 116 may notify therepresentative and/or their managers as part of the on-call guidance 145to engage in rapport building and pose questions at early stages of theconversation. Similarly, if one of the classifiers from the set ofclassifiers 140 indicates that matching speech rate to within 10% ofcustomer's rate at a specified relative position of the call, e.g.,during third minute of the call, leads to improved closing results, thecall-modeling component 116 may notify the representative and/or theirmanagers as part of the on-call guidance 145 to adjust their speech rateaccordingly. On the other hand, if one of the classifiers from the setof classifiers 140 indicates that conversations beginning with over aspecified number of objections, naming a specific competitor and mentionof the phrase “read online” do not lead to good results, thecall-modeling component 116 may notify the representative and/or theirmanagers as part of the on-call guidance 145 to expedite wrap-up ofconversations to avoid losing time on a call that is not likely to yielddesired results.

In addition to live on-call guidance, the real-time analysis component130 may be used to provide the representative and/or their managers withnon-real time analysis as well, which provides insight into details ofthe conversations, e.g., what occurred in the conversations, when eventsoccurred, and various such quantifiable analytics of the calls. Forexample, the classifiers can be used to find interesting calls thatwould interest the representatives to listen and learn from. Thedisclosed embodiments can be used to improve outcomes of the call notonly during a real-time or a live call, but could also be used to informrepresentatives and/or managers for better training and coaching inretrospect.

The real-time analysis component 130 may also be used to auto-populateinformation fields in a customer relationship management (CRM) system ora similar system.

FIG. 5 is a flow diagram of a process 500 for performing offlineanalysis of conversations between participants, consistent with variousembodiments. In some embodiments, the process 500 can be implemented inthe call-modeling system 100 of FIG. 1. At block 505, the offlineanalysis component 110 retrieves historical call data, e.g., call data105, regarding various conversations between participants, such as acustomer and a representative. In some embodiments, the call data 105can be audio recordings of calls between the participants, transcriptsof audio recordings, chat transcripts, etc. The offline analysiscomponent 110 can retrieve the call data 105 from the storage system125. Further, in some embodiments, the call data 105 can include dataregarding only a subset of the conversations stored in the storagesystem 125.

At block 510, the feature generation component 111 analyzes the calldata 105 to extract various features of the conversation, e.g., asdescribed at least with reference to FIGS. 1 and 2. Some examplefeatures include transcripts of audio recordings, vocabulary, semanticinformation of conversations, summarizations of utterances and variousnatural language entailments, voice signal associated features (e.g.,speech rate, speech volume, tone, and timber), emotions (e.g., fear,anger, happiness, timidity, fatigue), inter-speaker features (e.g.,similarity of speech rate between speakers, occurrence of firstinterrupt by customer, extrovert/introvert matching, or gender or ageagreement), personality traits (e.g., trustworthiness, engagement,likeability, dominance, etc.) and personal attributes (e.g., age,accent, and gender). The feature generation component 111 can alsoanalyze the call data 105 to generate various tags as described above.

At block 515, the classifier component 112 analyzes the features togenerate classifiers, e.g., as described at least with reference toFIGS. 1 and 3. The classifier component 112 analyzes the features 115using various techniques, e.g., machine learning algorithms such as SVM,DNN, to generate the classifiers 120. The classifiers 120 indicateconversation outcomes, e.g., “sales closed”, “sales failed,”“probability of recommending to a friend,” NPS, or customersatisfaction. Each of the classifiers indicates a specific outcome andcan include a set of the features that contributed to the specificoutcome. For example, in a sales call for renewing a magazinesubscription, a classifier “C1” can indicate that when laughter by acustomer and two open-ended questions from the representative areregistered, there is a high chance, e.g., 83%, of renewal. Theclassifier component 112 can generate multiple classifiers for the sameoutcome; however, they have distinct sets of features. Further, theclassifier component 112 generates different classifiers for differenttime windows of the conversations. For example, the classifier component112 generates a classifier “C1” for first two minutes of theconversations and a classifier “C2” for a third minute of theconversations. The offline analysis component 110 can store the features115 and the classifiers 120 in a storage system 125.

FIG. 6 is a flow diagram of a process 600 for modeling calls betweenparticipants to generate on-call guidance, consistent with variousembodiments. In some embodiments, the process 600 can be implemented inthe call-modeling system 100 of FIG. 1. At block 605, the real-timeanalysis component 130 receives real-time call data 150 of an ongoingconversation, e.g., an audio stream of a voice call between a customerand a representative. The ongoing conversation can be any of telephonebased, VoIP based, video conference based, VR based, AR based, e-mailbased, or based on any online meetings, collaborations or interactions.

At block 610, the feature generation component 113 analyzes thereal-time call data 150 to extract features, e.g., call features 135, ofthe ongoing conversation, e.g., as described at least with reference toFIGS. 1 and 2. The feature generation component 113 can also analyze thereal-time call data 150 to generate various tags as described above.

At block 615, the classifier component 114 inputs the extracted featuresto classifiers in the storage system, e.g., classifiers 120 which aregenerated as described at least with reference to process 500 of FIG. 5,to determine one or more classifiers that predict possible outcomes ofthe call based on the extracted features. For example, as described atleast with reference to FIGS. 1 and 4, the classifier component 114feeds the extracted features 135 into the classifiers 120 generated bythe offline analysis component 110, and selects a subset of theclassifiers 120, e.g., a set of classifiers 140, that includes featuresthat match with the call features 135 extracted from the liveconversation stream. In some embodiments, the set of classifiers 140include classifiers whose prediction power exceeds a specifiedthreshold. The set of classifiers 140 corresponds to specific outcomesand include real-time probabilities for the specific outcomes.

At block 620, the call-modeling component 116 generates on-callguidance, e.g., on-call guidance 145, that presents the real-timeprobabilities of possible outcomes of the call as indicated by the setof classifiers 140. The call-modeling component 116 can further analyzethe set of classifiers 140 to determine features that have highprediction power, e.g., prediction power exceeding a specifiedthreshold, for predicting a desired outcome, and then include thosefeatures and values associated with those features in the on-callguidance 145. The on-call guidance 145 notifies the representative toadopt or persist with an on-call behavior consistent with those featuresto achieve the desired outcome, or at least to increase the probabilityof achieving the desired outcome. For example, the on-call guidance 145can present instructions on a display screen of a user device associatedwith the representative recommending the representative to change therate of speech, use specific key words, or pose more open-endedquestions to the customer in order to increase the probability ofachieving the desired outcome.

Example Usage of the Embodiments

The following is an example usage of the disclosed embodiments formodeling sales calls for renewal of a subscription for a magazine. At afirst stage, e.g., before a call is received from a live customer orbefore a call is placed by a representative, a number of recordings ofprevious calls is processed by the offline analysis component 110, e.g.,using an ASR component 210 that is customized for the field of surgeryinstitutions, an NLP component 225, an affect component 215 and ametadata component 220 to generate various features. The classifiercomponent 112 generates two classifiers, based on those features, thatcan be found to be highly predictive: (a) a first classifier based onthe first two minutes of one or more of the analyzed conversations,which indicates that when a laughter by the customer is registered, therepresentative engages in rapport building, and at least two open-endedquestions are posed by the representative, then there is a high chance,e.g., 83%, of subscription renewal; (b) a second classifier based on thethird minute of one or more of the analyzed conversations, whichindicates that when a competitor magazine or the key-phrase “readonline” is used, and/or the speech rate of the customer is more thanthree words per second, the renewal chances drop to 10%.

The above two classifiers can then be used by the real-time analysiscomponent 130 in a second stage, e.g., during a live call between therepresentative and the customer, for generating an on-call guidance toguide the sales representatives as follows. Based on the firstclassifier, the real-time analysis component 130 can indicate to thesales representative to ask questions within the first 2 minutes. Basedon the second classifier, the real-time analysis component 130 can, atminute three of the conversation, urge the representative to reducespeech rate to get the customer to mirror their own speech rate if acompetitor is mentioned or otherwise the phrase “read online” is used.If the speech rate is not reduced, the real time analysis component 130can indicate to the representative and/or their managers to wrap up thecall as soon as possible.

The embodiments disclosed above may be implemented as separate modules,e.g., as presented above, as a single module, or any combinationthereof. Implementation details may vary, including core machinelearning algorithms employed. The embodiments may be implemented usingany software development environment or computer language. Theembodiments may be provided as a packaged software product, aweb-service, an API or any other means of software service. Theembodiments may use expert taggers, crowdsourcing or a hybrid approachfor tagging.

FIG. 7 is a block diagram of a computer system as may be used toimplement features of the disclosed embodiments. The computing system700 may be used to implement any of the entities, components or servicesdepicted in the examples of the foregoing figures (and any othercomponents described in this specification). The computing system 700may include one or more central processing units (“processors”) 705,memory 710, input/output devices 725 (e.g., keyboard and pointingdevices, display devices), storage devices 720 (e.g., disk drives), andnetwork adapters 730 (e.g., network interfaces) that are connected to aninterconnect 715. The interconnect 715 is illustrated as an abstractionthat represents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The interconnect 715, therefore, may include, for example,a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The memory 710 and storage devices 720 are computer-readable storagemedia that may store instructions that implement at least portions ofthe described embodiments. In addition, the data structures and messagestructures may be stored or transmitted via a data transmission medium,such as a signal on a communications link. Various communications linksmay be used, such as the Internet, a local area network, a wide areanetwork, or a point-to-point dial-up connection. Thus, computer readablemedia can include computer-readable storage media (e.g.,“non-transitory” media) and computer-readable transmission media.

The instructions stored in memory 710 can be implemented as softwareand/or firmware to program the processor(s) 705 to carry out actionsdescribed above. In some embodiments, such software or firmware may beinitially provided to the processing system 700 by downloading it from aremote system through the computing system 700 (e.g., via networkadapter 730).

The embodiments introduced herein can be implemented by, for example,programmable circuitry (e.g., one or more microprocessors) programmedwith software and/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more ASICs, PLDs, FPGAs, etc.

Remarks

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding of the disclosure. However, in someinstances, well-known details are not described in order to avoidobscuring the description. Further, various modifications may be madewithout deviating from the scope of the embodiments. Accordingly, theembodiments are not limited except as by the appended claims.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described, which may be requirementsfor some embodiments but not for other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, some termsmay be highlighted, for example using italics and/or quotation marks.The use of highlighting has no influence on the scope and meaning of aterm; the scope and meaning of a term is the same, in the same context,whether or not it is highlighted. It will be appreciated that the samething can be said in more than one way. One will recognize that “memory”is one form of a “storage” and that the terms may on occasion be usedinterchangeably.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for some terms are provided. A recital of one or moresynonyms does not exclude the use of other synonyms. The use of examplesanywhere in this specification including examples of any term discussedherein is illustrative only, and is not intended to further limit thescope and meaning of the disclosure or of any exemplified term.Likewise, the disclosure is not limited to various embodiments given inthis specification.

Those skilled in the art will appreciate that the logic illustrated ineach of the flow diagrams discussed above, may be altered in variousways. For example, the order of the logic may be rearranged, substepsmay be performed in parallel, illustrated logic may be omitted; otherlogic may be included, etc.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

We claim:
 1. A computer-implemented method comprising: receiving, by aprocessor, a stream of data related to a call in which a conversationoccurs between multiple participants; extracting, by the processor, aset of features from the stream of data, wherein the set of featuresincludes at least one characteristic of a given participant of themultiple participants and at least one characteristic of theconversation; analyzing, by the processor, the set of features usingmultiple classifiers to determine multiple potential outcomes of thecall; and generating, by the processor based on said analyzing, anon-call guidance to increase a probability of a given outcome of themultiple potential outcomes.
 2. The computer-implemented method of claim1, wherein the multiple participants includes (i) a customer of anenterprise and (ii) a representative of the enterprise.
 3. Thecomputer-implemented method of claim 1, wherein the multiple classifiersare identified based on multiple features extracted from historical datarelated to past calls.
 4. The computer-implemented method of claim 1,wherein said generating comprises: producing a notification thatindicates the given participant should adopt, desist, or persist with aconversation characteristic to increase or decrease the probability ofthe given outcome.
 5. The computer-implemented method of claim 4,wherein the conversation characteristic is associated with a pairing ofthe multiple participants.
 6. The computer-implemented method of 4,further comprising: determining, by the processor, that the set offeatures has changed as the call progressed; and adjusting, by theprocessor, the on-call guidance to reflect the change in the set offeatures.
 7. The computer-implemented method of claim 6, wherein saidadjusting comprises: producing a second notification that indicates thegiven participant should adopt a second conversation characteristicdifferent than the conversation characteristic.
 8. Thecomputer-implemented method of claim 1, wherein said generatingcomprises: analyzing the set of features with a subset of the multipleclassifiers to identify one or more features with a prediction powerexceeding a threshold, and presenting values of the one or more featuresas the on-call guidance.
 9. The computer-implemented method of claim 1,wherein said extracting comprises: generating the set of features byapplying a natural language processing algorithm, an artificialintelligence algorithm, a machine learning algorithm, or any combinationthereof to the stream of data.
 10. The computer-implemented method ofclaim 1, wherein the given participant is one of multiplerepresentatives responsible for communicating with customers on behalfof an entity, and wherein said analyzing comprises: generating themultiple classifiers based on multiple features extracted from a set ofcalls involving the multiple representatives, wherein each classifier ofthe multiple classifiers indicates an outcome of a conversationinvolving a given representative of the multiple representatives. 11.The computer-implemented method of claim 10, wherein said analyzingfurther comprises: analyzing audio recordings of the calls involving themultiple representatives to identify the multiple features.
 12. Thecomputer-implemented method of claim 1, wherein the set of featuresincludes a transcription, a vocabulary, and a language model of theconversation.
 13. The computer-implemented method of claim 12, whereinsaid extracting comprises: generating semantic information for theconversation based on the transcription, the vocabulary, and thelanguage model.
 14. The computer-implemented method of claim 1, whereinthe set of features includes data related to conversation flow.
 15. Thecomputer-implemented method of claim 1, wherein the set of featuresincludes an engagement metric that provides information regarding adegree of engagement of the given participant in the conversation. 16.The computer-implemented method of claim 1, wherein the set of featuresincludes multiple tags, each of which indicates a moment in the calldetermined to impact whether the given outcome occurs.
 17. Thecomputer-implemented method of claim 16, further comprising: sharing, bythe processor, the multiple tags with the given participant during thecall.
 18. The computer-implemented method of claim 1, wherein the callis a voice call, a video call, a virtual reality-based call, or anaugmented reality-based call.
 19. A non-transitory computer-readablemedium with instructions stored thereon that, when executed by aprocessor, cause the processor to perform operations comprising:receiving a stream of data related to a call in which a conversationoccurs between a customer and a representative; extracting a set offeatures from the stream of data, wherein the set of features includesat least one characteristic of the representative and at least onecharacteristic of the conversation; analyzing the set of features todetermine multiple potential outcomes of the call; and generating anon-call guidance to increase a probability of a given outcome of themultiple potential outcomes.
 20. The non-transitory computer-readablemedium of claim 19, wherein said analyzing comprises: applying multipleclassifiers to the set of features to identify the multiple potentialoutcomes of the call, wherein each classifier of the multipleclassifiers is associated with a different potential outcome of themultiple potential outcomes.
 21. The non-transitory computer-readablemedium of claim 19, wherein the on-call guidance presents probabilities,as determined in real time, of the multiple potential outcomes of thecall.
 22. A non-transitory computer-readable medium with instructionsstored thereon that, when executed by a processor, cause the processorto perform operations comprising: acquiring data related to past callsin which conversations occurred between customers and representatives ofan enterprise; extracting multiple features of the conversations byanalyzing the data; and generating multiple classifiers based on themultiple features, wherein each classifier of the multiple classifiersis associated with an outcome and at least one feature of the multiplefeatures determined to have contributed to the outcome.
 23. Thenon-transitory computer-readable medium of claim 22, wherein the dataincludes audio recordings of the conversations, transcripts of the audiorecords, transcripts of chat interactions, or any combination thereof.24. The non-transitory computer-readable medium of claim 22, wherein thefeatures include a transcription of each conversation, a vocabulary ofeach conversation, a language model of each conversation, semanticinformation for the conversations, voice signals for the conversations,emotion information for the conversations, inter-speaker features of theconversations, personality traits of the representatives, personalattributes of the representatives, or any combination thereof.
 25. Thenon-transitory computer-readable medium of claim 22, wherein at leasttwo classifiers of the multiple classifiers are associated with the sameoutcome despite being associated with different features.